Evaluating Long-term Dependency Benchmark Problems by Random Guessing

Numerous recent papers focus on standard recurrent nets' problems with tasks involving long-term dependencies. We solve such tasks by random weight guessing (RG). Although RG cannot be viewed as a reasonable learning algorithm we nd that it often outperforms previous, more complex methods on widely used benchmark problems. One reason for RG's success is that the solutions to many of these benchmarks are dense in weight space. An analysis of cases in which RG works well versus those in which it does not can serve to improve the quality of benchmarks for novel recurrent net algorithms.